Transmission apparatus operation for vpn optimization by defragmentation and deduplication  method

ABSTRACT

A transmission apparatus operation method for optimizing a virtual private network operates by defragmenting and de-duplicating transfer of variable sized blocks. A large data object is converted to a plurality of data paragraphs by a fingerprinting method. Each data paragraph is cached and hashed. The hashes are transmitted to at least one satellite apparatus. Only data paragraphs which were not previously cached at each satellite are transferred.

RELATED PATENTS AND APPLICATIONS

This application is a division of application Ser. No. 13/166,808 filedon Jun. 23, 2011, which was a division of VPN OPTIMIZATION BYDEFRAGMENTATION AND DEDUPLICATION APPARATUS AND METHOD Ser. No.12/421,651 filed Apr. 10, 2009. Priority is claimed from the filing dateof the above application which issued on Nov. 1, 2011 as U.S. Pat. No.8,050,251.

BACKGROUND

1. Technical Field

The present invention relates generally to network communications overTCP/IP and more particularly to connecting low bandwidth servicesbetween local area networks (LANs) and ameliorating packetfragmentation.

2. Description of Related Art

It is known that virtual private networks (VPN) allow remote employeesaccess to an enterprise's information systems. VPNs are used to connectremote offices to headquarters for time critical enterprise resourcemanagement operations.

The communication network typically comprises a public network (e.g.,the Internet). The connections to the communication network from thebranch office and the central office typically cause a bandwidthbottleneck for exchanging the data over the communication network. Theexchange of the data between the branch office and the central office,in the aggregate, will usually be limited to the bandwidth of theslowest link in the communication network aggravated by the latencyimposed by encryption and decryption of the VPN overhead.

For example, the router connects to the communication network by a T1line, which provides a bandwidth of approximately 1.544 Megabits/second(Mbps). The router 170 connects to the communication network by a T3line, which provides a bandwidth of approximately 45 Megabits/second(Mbps). Even though the communication network may provide an internalbandwidth greater than 1.544 Mbps or 45 Mbps, the available bandwidthbetween the branch office and the central office is limited to thebandwidth of 1.544 Mbps (i.e., the T1 connection).

Moreover, many applications do not perform well over the communicationnetwork due to the limited available bandwidth. Developers generallyoptimize the applications for performance over a local area network(LAN) which typically provides a bandwidth between 10 Mbps toGigabit/second (Gbps) speeds. The developers of the applications assumesmall latency and high bandwidth across the LAN between the applicationsand the data. However, the latency across the communication networktypically will be 100 times that across the LAN, and the bandwidth ofthe communication network will be 1/100th of the LAN.

Connecting a branch office to headquarters is likely to involve tyingtwo local area networks to routers which are connected by a wide areanetwork. This requires traversing a number of gateways controlled bydifferent parties. The maximum packet size (also called the MTU, orMaximum Transmission Unit) and default packet size can vary depending onthe media. For ethernet (LAN), the max packet size is 1500 octets. Fortoken ring and FDDI, it is 4096 octets. The IP protocol was designed foruse on a wide variety of transmission links. Although the maximum lengthof an IP datagram is 64K, most transmission links enforce a smallermaximum packet length limit, called an MTU. The value of an MTU dependson the type of the transmission link.

The design of IP accommodates MTU differences by allowing routers tofragment IP datagrams as necessary. The receiving station is responsiblefor reassembling the fragments back into the original full size IPdatagram. As the IP packets are routed independently of each other,different packets between the same end hosts could take different routeswith varying MTU sizes. However, the lack of end-to-end information canquickly result in oversized packets being received by the intermediaterouters that have to route them somehow.

The IP protocol provides a convenient solution: the IP fragmentation, amechanism where a single inbound IP datagram is split into two or moreoutbound IP datagrams. The worst impact of IP fragmentation is in therouter-to-router communication. If a router-to-router IP packet isfragmented somewhere in the path, the receiving router has to reassemblethe original packet, resulting in significantly reduced switchingperformance.

An additional problem with deployment of VPNs is that there is latencyintroduced by the encryption and decryption of transmissions. Because ofthe encryption of traffic, the same files transmitted twice will notlook the same and this prevents conventional caching strategies.

For example, in a centralized server implementation having multiplebranches, computers in each of the multiple branch offices make requestsover the VPN to central servers for the organization's data. The datatransmitted by the central servers in response to the requests quicklysaturate the available bandwidth of the central office's connection tothe communication network, further decreasing application performanceand data access at the multiple branch offices. This is particularlytroublesome for entities which span multiple timezones as congestion candominate the work day.

It is also known that mechanisms for caching improve applicationperformance and data access. A cache is generally used to reduce thelatency of the communication network (e.g., communication network)forming the VPN (i.e., because the request is satisfied from the localcache) and to reduce network traffic over the VPN (i.e., becauseresponses are local, the amount of bandwidth used is reduced).

Webpage caching, for example, is the caching of web documents (i.e.,HTML pages, images, etc.) in order to reduce web site access times andbandwidth usage. Web caching typically stores local copies of therequested web documents. The web cache satisfies subsequent requests forthe web documents if the requests meet certain predetermined conditions.

One problem with web caching is that the Time to Live parameter isgenerally not easily changed. Thus the management of a web cache is atleast tricky and not conveniently purged or updated. Every browser canhave a slightly different version of a document. Another problem is thatthe web cache stores entire objects (such as documents) and cache-hitsare binary: either a perfect match or a miss. Even where only smallchanges are made to the documents, the web cache cannot use the cachedcopy of the documents to reduce network traffic.

It is also known that randomly chosen polynomials are used to“fingerprint” bit-strings. This method, first published by Michael O.Rabin Center for Research in Computing Technology Harvard UniversityReport TR-15-81 (1981), is applied to produce a very simple stringmatching algorithm and a procedure for securing files againstunauthorized changes. The method is provably efficient and highlyreliable. However it is also known that the Rabin fingerprinting schemeis not as secure as more expensive cryptographic hash functions.

It is known that the Rabin-Karp algorithm is a string searchingalgorithm created by Michael O. Rabin and Richard M. Karp in 1987 thatuses hashing to find a substring in a text. It is used for multiplepattern matching rather than single pattern matching. Running timeperformance is considered a reason that it is not widely used. However,it has the advantage of being able to find any one of kstrings or lessin a predictable time regardless of the magnitude of k.

REFERENCES

-   U.S. Pat. Nos. 5,511,159, 5,627,748, 5,778,231, 5,953,006, and-   U. Manber, “Finding Similar Files In a Large File System”, Proc.    1994 Winter Usenix Technical Conference, January 1994, pp. 1-10.-   B. S. Baker, “Parameterized Pattern Matching: Algorithms and    Applications,” J. Comput. Syst. Sci. 52(1), February 1996, pp.    28-42.-   B. S. Baker, “Parameterized Duplication In Strings: Algorithms and    an Application to Software Maintenance,” SIAM J. Computing, 26(5),    October 1997, pp. 1343-1362.-   E. W. Myers, “An O(ND) Difference Algorithm and Its Variations,”    Algorithmica, 1986, pp. 1:251-266.-   B. S. Baker, “On Finding Duplication and Near-duplication in Large    Software Systems,” Second Working Conference on Reverse Engineering,    1995, pp. 86-95.-   H. L. Berghel and D. L. Sallach, “Measurements of Program Similarity    in Identical Task Environments,” SIGPLAN Notices, 9(8), August 1984,    pp. 65-76.-   S. Brin, J. Davis, and H. Garcia-Molina, “Copy Detection Mechanisms    For Digital Documents,” Proceedings of the ACM Special Interest    Group on Management of Data (SIGMOD), 1995, pp. 1-21.-   A. Broder, S. Glassman, M. Manasse, and G. Zweig, “Syntactic    Clustering of the Web,” Proceedings of the Sixth International World    Wide Web Conference, April 1997, pp. 391-404.-   K. W. Church and J. I. Helfman, “Dotplot: A Program For Exploring    Self-similarity In Millions of Lines of Text and Code,” Journal of    Computational and Graphical Statistics, 2(2), June 1993, pp.    153-174.-   N. Heintz, “Scalable Document Fingerprinting,” Proceedings of the    Second USENIX Workshop on Electronic Commerce, Nov. 18-21, 1996, pp.    1-10.-   S. Horwitz, “Identifying the Semantic and Textural Differences    Between Two Versions of a Program,” Proceedings of the ACM SIGPLAN    Conference on Programming Language Design and Implementation (PLDI),    June 1990, pp. 234-245.-   H. T. Jankowitz, “Detecting Plagiarism in Student PASCAL Programs,”    Computer journal, 31(1), 1988, pp. 1-8.-   J. H. Johnson, “Substring Matching For Clone Detection and Change    Tracking,” Proc. International Conf. on Software Maintenance, 1994,    pp. 1-7.-   PocketSoft . . . RTPatch Professional, Feb. 23, 1998-   T. Proebsting and S. A. Watterson, Krakatoa: Decompilation in Java    (does bytecode reveal source:). USENIX Conference on Object-oriented    Technologies and Systems, June 1997, pp. 1-13.-   N. Shivakumar and H. Garcia-Molina, “Building a Scalable and    Accurate Copy Detection Mechanism,” Proceedings of 1st ACM    International Conference on Digital Libraries (DL'96), March 1996,    pp. 1-9.

On finding duplication in strings and software, technical report, AT&TBell Laboratories, February, 1993

-   Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein,    Clifford (Sep. 1, 2001). “The Rabin-Karp algorithm”. Introduction to    Algorithms (2nd edition ed.). Cambridge, Mass.: MIT Press. pp.    911-916. ISBN 978-0262032933.

It is also known that a hash function is any well-defined procedure ormathematical function which converts a large, possibly variable-sizedamount of data into a small datum. The values returned by a hashfunction are called hash values, hash codes, hash sums, or simplyhashes. In the present patent application we define a variable-sizedamount of data converted to a hash as a data paragraph. A circuit isdisclosed for selecting data paragraphs from a data object. That portionof a data object which is below the minimum size of a data paragraph isdefined as a remainder.

It is known that hash functions are used to speed up table lookup ordata comparison tasks—such as finding items in a database, detectingduplicated or similar records in a large file, finding similar stretchesin DNA sequences, and so on.

Thus it can be appreciated that connecting branch offices withenterprise applications presents bandwidth, security, and data integrityproblems which are aggravated by virtual public networks. What is neededis a way to address VPN fragmentation, data duplication and enable lowlatency, high responsiveness for users who must live remotely through anencrypted, low bandwidth link from their central datacenter andapplications.

SUMMARY OF THE INVENTION

A first apparatus divides a data object into one or more data paragraphsof minimum size and a remainder if the minimum size requirement for adata paragraph is not met. Each data paragraph is submitted to a hashfunction. On the condition that the hash has not previously been sent toor received from a destination, the paragraph is transmitted to thedestination. On the condition that all necessary data paragraphs arecached at the second apparatus, the first apparatus transmits the hashesand if necessary the remainder which the second apparatus uses toreconstitute the data object.

In an embodiment, the second apparatus receives a list of the hashes fordata paragraphs pending transfer and updates the first apparatus onunnecessary transfers to optimize the VPN traffic.

Each data paragraph is determined by sequentially computing a low-costcumulative fingerprint on byte boundaries of the data object andstopping on one of two conditions: that the fingerprint matches a maskor that the data paragraph has reached a maximum desired size. Thefingerprint is discarded and more expensive hash is computed and savedto identify the data paragraph.

The invention addresses the above problems by providing a VPNdefragmentation and de-duplication apparatus, method, and softwareproduct. A first apparatus includes a communication interface configuredto communicate with a communication network and a processor. Theprocessor receives a data object and converts it to data paragraphs eachof which has a hash value. The apparatus generates a packet containingall the hashes and transmits it to a second apparatus through acommunications network.

Advantageously, the first apparatus does not transfer a data paragraphover the communication network if the data paragraph is cached at thesecond apparatus. The first apparatus effectively reduces latency overthe communication network, and reduces network traffic by minimizing theamount of data sent over the communication network. As a result, theeffective maximum transmission unit (MTU) for remote office traffic canapproach that of local networked users connected by Ethernet.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a computer.

FIG. 2 is a block diagram of portions of a satellite apparatus.

FIG. 3 is a block diagram of portions of a satellite apparatus.

FIG. 4 is a block diagram of portions of a satellite apparatus and aprimary apparatus.

FIG. 5 is a block diagram of portions of a primary apparatus.

FIG. 6 is a block diagram of a system of a satellite apparatus and aprimary apparatus.

FIG. 7 is a block diagram of a best mode of a satellite apparatus.

FIG. 8 is a block diagram of a best mode of a primary apparatus.

FIG. 9 is a block diagram of a system of two apparatus operating over awide area network.

FIG. 10 is flow chart of a method for defragmenting and de-duplicating.

DETAILED DISCLOSURE OF EMBODIMENTS

The embodiments discussed herein are illustrative of one example of thepresent invention. As these embodiments of the present invention aredescribed with reference to illustrations, various modifications oradaptations of the methods and/or specific structures described maybecome apparent to those skilled in the art. All such modifications,adaptations, or variations that rely upon the teachings of the presentinvention, and through which these teachings have advanced the art, areconsidered to be within the scope of the present invention. Hence, thesedescriptions and drawings should not be considered in a limiting sense,as it is understood that the present invention is in no way limited toonly the embodiments illustrated.

To provide improved application performance and data access, the VPNdefragmentation system generally comprises a first apparatus and asecond apparatus. The first apparatus receives a data object andconverts it to a plurality of hashes which corresponds to dataparagraphs stored in cache. Some of the data paragraphs must betransferred to the second apparatus and others not.

The VPN defragmentation and de-duplication system provides that thesecond apparatus processes the hashes and reconstitutes the originaldata object and transmits it to the original destination. Thus, the VPNdefragmentation and de-duplication system operates to reduce latency andnetwork traffic over the communication network.

FIG. 1 shows a block diagram of a typical computing system 100 where thepreferred embodiment of this invention can be practiced. The computersystem 100 includes a computer platform having a hardware unit 103, thatimplements the methods disclosed below. The hardware unit 103 typicallyincludes one or more central processing units (CPUs) 104, a memory 105that may include a random access memory (RAM), and an input/output (I/O)interface 106. Microinstruction code 107, may also be included on theplatform 102. Various peripheral components may be connected to thecomputer platform 102. Typically provided peripheral components includean external data storage device (e.g. flash, tape or disk) 110 where thedata used by the preferred embodiment is stored. A link 112 may also beincluded to connect the system 100 to one or more other similar computersystems. The link 112 may also provide access to the global Internet. Anoperating system (OS) 114 coordinates the operation of the variouscomponents of the computer system 100, and is also responsible formanaging various objects and files, and for recording certaininformation regarding same. Lying above the OS 114 is an application andsoftware tools layer 114A containing, for example, compilers,interpreters and other software tools. The interpreters, compilers andother tools in the layer 114A run above the operating system and enablethe execution of programs using the methods known to the art.

An example of a suitable CPU is a Xeon™ processor (trademark of theIntel Corporation); examples of an operating systems is GNU/Linux;examples of an interpreter and a compiler are a Perl interpreter and aC++ compiler. Those skilled in the art will realize that one couldsubstitute other examples of computing systems, processors, operatingsystems and tools for those mentioned above. As such, the teachings ofthis invention are not to be construed to be limited in any way to thespecific architecture and components depicted in FIG. 1.

Referring now to FIG. 2 a large data object is received by circuit 210to generate a plurality of data paragraphs, hashes, and a remainder. Thedata paragraphs are variable in size between a minimum size and amaximum size. In an embodiment the data paragraphs are fingerprinted andwhen the fingerprint matches a mask a variable size length block istransferred to a circuit to compute a hash value. The hash and the dataparagraph are stored in a satellite cache store 231 if the hash is notalready stored there. The hash and the reception apparatus address arestored in the satellite transfer diary store 222 with “pending” fortransfer date if the hash and destination apparatus are not alreadystored there. If the hash and the destination apparatus are alreadystored there, the transfer date is updated to the present time. As eachhash is computed it is transferred to a circuit 243 for packing hashesand a remainder into a data object integration packet along with thedestination of the data object. It is understood that a receptionapparatus may cache a data paragraph which will be required for severaldata objects and/or being transferred to more than one TCP/IPdestination.

Referring to FIG. 3, as each hash record is stored or updated into thesatellite transfer diary store 222, a circuit 234 for transmittingpending transfers queues transfers from the satellite cache store 231 tothe primary cache store 131. In an embodiment, illustrated in FIG. 4,the circuit 234 to transmit pending transfer also sends a list ofpending transfers to a circuit to receive pending transfer. In anembodiment, the circuit 136 to receive a pending transfer checks aprimary transfer diary store 122 to determine if the hash is alreadystored in the primary store cache. Under the condition that a hash isalready stored in the primary cache store 131, the circuit to receivepending transfer updates the status of the satellite transfer diarystore 222 from pending to past whereby a data paragraph transfer isremoved from the queue of the circuit for pending transfer. When thecircuit to transmit a pending transfer completes a transfer, the statusin the satellite transfer diary store 222 is updated from pending topast. In an embodiment the current date-time represents the past and all1's represents pending.

Upon the condition that a data object can provide no further dataparagraphs and thus no further hashes the rest of the data object istransferred as a remainder to a circuit 243 for packing a data objectintegration packet.

Referring to FIG. 5, upon the condition that all pending transfers for aspecific data object have been completed, a circuit 258 for determiningthat all hashes related to a data object are located in the primarycache store enables the transmission of the data object integrationpacket to the destination.

A circuit 190 to integrate data objects retrieves the data paragraphsspecified by hashes in the data object integration packet. To this isconcatenated the remainder thus integrating the original data object.

FIG. 6 illustrates a first satellite apparatus and a primary apparatus.It is understood and appreciated that a plurality of satellites issupported by a primary apparatus.

FIG. 7 is a block diagram of a satellite apparatus. FIG. 8 is a blockdiagram of a primary apparatus. The primary transfer diary store recordsall transfers between the primary and any satellite. A satellitetransfer diary store records the transfers between itself and itsprimary. In an embodiment, some or all of the data paragraphs stored atthe primary cache store are distributed in anticipation of future use atselected or all satellites. In an embodiment, purging of outdated dataparagraphs may be synchronized from the primary to any one or all of thesatellites.

FIG. 9 illustrates an embodiment of the invention connecting a primaryapparatus to a satellite apparatus through a plurality of persistentconnection operating a Virtual Private Network over a Wide Area Network.The TCP connection between the primary and satellite opens large windowsthrough the VPN system. The large windows enable transfers of variableblocks from one cache to another as well as transfers of the hasheswhich reference variable blocks stored in each cache with minimum TCP/IPfragmentation.

FIG. 10 illustrates a method of operating the apparatus. It isunderstood that a circuit may be embodied as a processor under controlof software adapted to perform certain operations. The processes thatcontrol a processor to perform the operations of the claimed inventionare: terminating a local TCP connection, defragmenting a data objecttransmitted in a plurality of small TCP packets, variable blocking ofthe data object, computing a hash for each variable block, querying astore for prior caching of the variable block, packing a packet with thehashes computed from a data object and the remainder of the data objectwhich is not cached, and sending those variable blocks to thedestination apparatus not already at the destination apparatus followedby sending the packet with the references. At least one TCP connectionwith large window is established between the source apparatus anddestination apparatus.

The disclosed invention is an apparatus comprising a circuit to generatea data paragraph, a hash, and a remainder;

a satellite cache store;

a satellite transfer diary store;

and a circuit to pack a plurality of hashes and a remainder into apacket.

The method for operating the above apparatus is

receiving a data object;

generating at least one data paragraph

computing a hash for each data paragraph;

determining a remainder;

storing a data paragraph into a satellite cache store if the hash isdetermined to be new;

storing a record into a satellite transfer diary of the destination foreach data paragraph, the hash, and the status for transferring the dataparagraph to the destination; and

transferring at least one hash and a remainder to a circuit to pack adata object integration packet.

The apparatus further comprises

a circuit for transmission of a pending transfer; and

a circuit for to determine that data paragraphs corresponding to hashesare located at the primary cache store; and

at least one network interface.

The method further comprises the steps

reading a pending status for transfer of a data paragraph;

transferring a data paragraph from a satellite cache store to a primarycache store;

updating the status of the record in satellite transfer diary store frompending to past;

determining that all data paragraphs required for data objectintegration are stored at the primary cache store; and

enabling transmission of a data object integration packet to thedestination.

In an embodiment of the apparatus, the circuit to generate a dataparagraph, a hash, and a remainder is coupled to the satellite cachestore whereby a data paragraph is stored corresponding to each hash, andthe circuit to generate a data paragraph, a hash, and a remainder iscoupled to the satellite transfer diary store whereby the destination,the hash, and the status of transferring the data paragraph to thedestination is recorded wherein status comprises a past date or pending,and the circuit to generate a data paragraph, a hash, and a remainder iscoupled to a circuit to pack hashes and a remainder into a data objectintegration packet addressed to a destination, whereby duplicativetransfer of a data paragraph is avoided if is already stored in thedestination cache.

In an embodiment of the apparatus, the circuit for to generate a dataparagraph, a hash, and a remainder comprises a circuit to select astring between a minimum length and a maximum length by computing afingerprint and matching selected bits of the fingerprint to a certainmask by reading from the beginning of a data object or from the end ofthe previous data paragraph.

In an embodiment of the apparatus, the circuit to generate a dataparagraph, a hash, and a remainder comprises a circuit to compute anSHA1 value on a variable length string. In an embodiment of theapparatus the circuit to generate a data paragraph, a hash, and aremainder comprises a circuit to compute an MD5 value on a variablelength string. It is understood that other hash functions may beemployed within the scope of the invention but that well known hashfunctions are the best mode presently adopted.

The method further comprises steps within the circuit for to generate adata paragraph, a hash, and a remainder comprising computing afingerprint and determining a data paragraph on the occurrence of acertain string which may include wildcard characters being matched by afingerprint or a maximum length being reached. A minimum size for a dataparagraph is selected for efficiency and matching the fingerprint willnot be done below the minimum size.

The method further comprises steps within the circuit for to generate adata paragraph, a hash, and a remainder comprising determining a dataparagraph by using digital signal processing detection of a repeatingpattern. In transmitting a two dimensional data object such as a blockdiagram, a edge detection or line detection operation within a digitalsignal processor would trigger a data paragraph delimitation.

The method further comprises steps within the circuit for to generate adata paragraph, a hash, and a remainder comprising selecting a pluralityof data paragraphs from the start of a data object and on the conditionthat no more data paragraphs can be selected, assigning the rest of thedata object as a remainder and transferring the hashes computed for eachdata paragraph and the remainder to a circuit for packing a data objectintegration packet addressed to the destination of the data object.

An apparatus for receiving defragmented and de-duplicated data objectpackets comprises

at least one network interface;

a primary cache store;

a primary transfer diary store; and

a data object integrator.

The method for operating the above apparatus comprises

receiving and storing into primary cache store at least one dataparagraph corresponding to a hash,

storing into primary transfer diary store the source apparatus of theparagraph corresponding to the hash and the time of most recent usage,

receiving a data object integration packet containing at least one hashand a remainder; and

integrating a data object by reading data paragraphs from the primarycache store and concatenating the data paragraphs and the remainder andtransferring the reintegrated data object to its original destinationlocal to the receiving apparatus.

The apparatus further comprises

a circuit to receive a pending transfer;

the circuit to received a pending transfer coupled to a circuit for totransmit a pending transfer;

the circuit to receive a pending transfer coupled to a primary transferdiary store; and

the circuit to receive a pending transfer coupled to a satellitetransfer diary store.

The method further comprises the steps

within a circuit to receive a pending transfer:

receiving a hash from a pending transfer transmitter corresponding to adata paragraph;

determining from a primary transfer diary store that the hash is alreadystored in a primary cache store; and

transmitting to a satellite transfer store an update on the status ofthe hash;

whereby the satellite transfer diary store changes the status of apending transfer from pending to past and utilization of acommunications channel is optimized by de-duplication.

The principles discussed herein are equally applicable to multiplesatellites (not shown) and to multiple primaries (not shown). Forexample, the VPN defragmentation system may include multiple branchoffices and/or multiple central offices coupled to the communicationnetwork. Branch office/branch office communication and centraloffice/central office communication, as well as multi-apparatus and/ormulti-node communication and bi-directional communication are furtherwithin the scope of the disclosure. However, for the sake of simplicity,the disclosure provides the non-limiting illustrative exemplary VPNdefragmentation and deduplication system having the single satellite andthe single primary, and the respective communication network couplingthem.

A communication network comprises hardware and/or software elements thatenable the exchange of information (e.g., voice and data) between asatellite and a primary. Some examples of the communication network areSIPRNET, JWICS, and the Internet. Typically connections from the branchoffice to the communication network are ISDN, T1 lines (1.544 Mbps), andpossibly broadband connections such as digital subscriber lines (DSL)and cable modems.

It will be understood that a satellite apparatus and a primary apparatusperform bidirectional communication. For example, a data paragraph sentto the primary apparatus from the satellite apparatus is be stored in alocation locally accessible to the primary apparatus and in a locationlocally accessible to the satellite apparatus. If the data paragraph isto be transferred again from either one to the other, the respectivetransfer diary store determines that that the pattern is locallyaccessible and that an additional transfer is unnecessary. The sendingapparatus transmits the hash to the destination apparatus whichreconstitutes the data object from its cache. Therefore, each apparatusin the VPN defragmentation and de-duplication system advantageously usesdata paragraphs previously transferred to and from the apparatus toreduce network traffic with other apparatuses in the VPN defragmentationand de-duplication system.

A circuit 210 calculates a fingerprint at every byte boundary of a dataobject to be sent across the communication network. The fingerprints areinfluenced by preceding bytes in the data flow. For example, thefingerprints are influenced by approximately the n previous bytes, wheren determines the fingerprint. In an example 4000 bytes. Some examples ofcalculating include Rabin's fingerprints and checksums over the previousn bytes of the data flow. In some embodiments, rolling implementationsof fingerprints are used where a new byte is added, and a byte from nbytes earlier is removed. When a fingerprint matches a selected triggervalue such as bitmask with wildcard characters, a hash is computed andcompared with previous hash values stored in a local database.

While hashing and fingerprinting are sometimes confused in theliterature because of their similarity, within this patent applicationwe distinguish a fingerprint to be a low cost, efficient computationwhich allows some risk of collision and a hash to be a more costlycomputation with proven and quantifiable low risk of collision. The hashvalue uniquely identifies its data paragraph.

In an embodiment an apparatus includes a processor, a memory, a VPNcommunication interface, a LAN communication interface, and a transferdiary store. A system bus links the processor, the memory, the VPNcommunication interface, the LAN communication interface, and thetransfer diary store. A high capacity cache store is attached.

The transfer diary store comprises hardware and/or software elementsconfigured to store data in an organized format to allow a processor tocreate, modify, and retrieve the data. The transfer diary store mayorganize the data to enable the determination of whether a dataparagraph is locally accessible to an apparatus, and to enable quickretrieval of locally accessible data to the apparatus. The hardwareand/or software elements of the diary store may include storage devices,such as RAM, hard drives, optical drives, flash memory, and magnetictape. In some embodiments, the apparatus implements a virtual memorysystem with linear addresses, the locally accessible data, and largeaddress space.

In an embodiment of the invention disclosed above, a method is disclosedfor operating a system to optimize a virtual private network bydefragmentation and de-duplication of large data transmissions;

the system comprising:

a wide area network coupling a satellite apparatus to a primaryapparatus;

a satellite apparatus comprising:

a circuit to receive a data object and determine a plurality of dataparagraphs;

a circuit to compute a hash for each data paragraph;

a circuit to pack a packet with a plurality of hashes;

a primary apparatus comprising:

a data paragraph cache store;

a circuit to receive a packed packet from a satellite apparatus;

a circuit to integrate data paragraphs into a data object,

and

network interfaces and means for sending and receiving TCP packets ineach apparatus; the method comprising the steps following

opening a virtual private network connection between a satelliteapparatus and a primary apparatus;

establishing a persistent TCP connection between a satellite apparatusand a primary apparatus through a wide area network;

setting a maximum TCP buffer size at both the satellite apparatus andthe primary apparatus;

setting a TCP window scale to support large transmission sizes;

transmitting at least one packed packet comprising a plurality of hashesand a destination IP address between a satellite apparatus and a primaryapparatus;

within the primary apparatus:

receiving data paragraphs and hashes from a satellite apparatus untilall data paragraphs are available;

receiving a packed packet comprising a plurality of hashes and adestination IP address;

integrating a data object by retrieving data paragraphs from dataparagraph cache store; and

transmitting the data object to the destination IP address;

within the satellite apparatus:

receiving a data object and a destination IP address;

determining a plurality of data paragraphs by computing fingerprints onthe data object;

packing a packet with hashes computed from each data paragraph and withthe destination IP address;

checking a transfer diary to determine if data paragraphs are availablein the primary cache;

queuing data paragraphs for transmission to primary cache;

transmitting a hash to a primary apparatus for each data paragraphqueued for transmission to said primary apparatus;

determining that all data paragraphs required for a data object areavailable at the primary apparatus; and

transmitting the packed packet to the primary apparatus.

The method referred to above of determining a plurality of dataparagraphs by computing fingerprints on the data object comprises thefollowing steps within the satellite apparatus:

incrementally reading a data object as a string of text;

delimiting a data paragraph after a minimum length;

delimiting a data paragraph at a maximum length;

delimiting a data paragraph upon matching a fingerprint to a masked keyprior to the maximum length;

computing the fingerprint upon reading each additional byte of the dataobject after a minimum length and prior to a maximum length; and

delimiting a data paragraph upon reaching the end of the data object.

The method referred to above of receiving data paragraphs and hashesfrom a satellite apparatus until all data paragraphs are available,comprises the steps within the primary apparatus of:

receiving a hash from a satellite apparatus;

checking a diary store to determine if a data paragraph having a hash isavailable;

receiving and storing data paragraphs into data paragraph cache store ifneeded; and

signaling to a satellite apparatus when all queued data paragraphs areavailable in data paragraph cache store.

The disclosed method further comprises the steps of

receiving a plurality of packets having a common destination IPcorresponding to fragmentation of a data object intended to betransmitted through a virtual private network from a source in a localarea network coupled to the satellite apparatus to a destination IPaddress in a local area network coupled to the primary apparatus;

terminating the connection; and

defragmenting the data object.

In an embodiment the method is improved by the steps of

determining that a data paragraph queued for transmission from asatellite apparatus to a primary apparatus is unnecessary and

signaling the satellite apparatus to remove it from the transmissionqueue.

The apparatus further comprises:

circuits to couple to a local area network attached to the satelliteapparatus, and to a local area network attached to the primaryapparatus, and to a wide area network attached to both the satelliteapparatus and the primary apparatus;

a circuit to receive and terminate a TCP connection from a source IPaddress at the satellite apparatus;

a circuit to initiate a TCP connection to a destination IP address atthe primary apparatus; and

a circuit to open a virtual private network between the satelliteapparatus and the primary apparatus, open a persistent TCP connection,set TCP buffer size, set TCP window scale, and negotiate maximum TCPwindow size.

The apparatus further comprises

a circuit to receive and defragment a plurality of packets containingdata being transmitted to a destination IP address and combine them intoa data object within the satellite apparatus, and a circuit to fragmenta data object integrated at a primary apparatus from a plurality of dataparagraphs into a plurality of packets and transmit them to adestination IP address.

The apparatus further comprises

a circuit to divide a data object into a plurality of data paragraphs;

a circuit to compute a hash for each data paragraph using one of MD5 orSHA1;

a circuit to pack a packet with the destination IP address for a dataobject and the hashes of the data paragraphs the data object was dividedinto and to transmit the packed packet to a primary apparatus;

a circuit to receive a packed packet, read data paragraphs from a dataparagraph cache store, integrate the data paragraphs into a data objectat the primary apparatus; and

a data paragraph cache store at each of the satellite apparatus and theprimary apparatus.

The apparatus further comprises

a circuit to determine if a data paragraph has previously beentransmitted to the primary apparatus by the satellite apparatus, queuethe data paragraph for transmission, and enable the transmission of thepacked packet of hashes upon the condition that the data paragraphs areavailable at the primary apparatus;

a circuit to transmit data paragraphs to the primary apparatus; and

a circuit to determine if a data paragraph is available at the primaryapparatus and remove it from the transmission queue.

It is known that a circuit embodiment is at least one processor adaptedby computer readable instructions encoded on media to perform the steps:

perform VPN and TCP operations;

delimit a data object into data paragraphs by fingerprinting;

compute hashes for data paragraphs;

pack and transmit a packet containing hashes;

determine if a data paragraph needs to be transmitted, and when it isavailable;

store and read data paragraphs into and out of data paragraph cachestore;

integrate data paragraphs into data objects; and

transmit a plurality of packets containing a data object to adestination IP; and

means for connecting said circuit, cache store, and network interfaces.

An embodiment of the invention comprises a method for operating atransmission apparatus coupled to a local area network and coupled to awide area network, the apparatus comprising:

a circuit to generate a data paragraph, a hash, and a remainder;

a satellite cache store;

a satellite transfer diary store; and

a circuit to pack a plurality of hashes and a remainder into a packet;the method comprising:

receiving a data object;

generating at least one data paragraph;

computing a hash for each data paragraph;

determining a remainder;

storing a data paragraph into a satellite cache store if the hash isdetermined to be new;

storing a record into a satellite transfer diary of the receptionapparatus for each data paragraph, the hash, and the status of thetransfer of the data paragraph to the reception apparatus; and

transferring at least one hash and a remainder to a circuit to pack adata object integration packet.

The transmission apparatus of the invention comprises

a circuit for transmission of a pending transfer;

a circuit for to determine when data paragraphs corresponding to hashesare located at primary cache store; and

at least one network interface.

The method of the invention further comprises the steps within thetransmission apparatus:

reading a pending status for transfer of a data paragraph;

transferring a data paragraph from a satellite cache store to a primarycache store;

updating the status of the record in satellite transfer diary store frompending to past;

determining that all data paragraphs required for data objectintegration are stored at the primary cache store; and

enabling transmission of a data object integration packet to thereception apparatus.

In a transmission apparatus the circuit to generate a data paragraph, ahash, and a remainder is coupled to the satellite cache store whereby adata paragraph is stored corresponding to each hash, and the circuit togenerate a data paragraph, a hash, and a remainder is coupled to thesatellite transfer diary store whereby the reception apparatus, thehash, and the status of the transfer of the data paragraph to thereception apparatus is recorded wherein status comprises a past date orpending, and the circuit to generate a data paragraph, a hash, and aremainder is coupled to a circuit to pack hashes and a remainder into adata object integration packet addressed to a reception apparatus,whereby duplicative transfer of a data paragraph is avoided if it isalready stored at the reception apparatus.

In the transmission apparatus the circuit to generate a data paragraph,a hash, and a remainder comprises a circuit to select a string between aminimum length and a maximum length by computing a fingerprint andmatching selected bits of the fingerprint to a certain mask by readingfrom the beginning of a data object or from the end of the previous dataparagraph.

In an embodiment of the transmission apparatus, the circuit to generatea data paragraph, a hash, and a remainder comprises a circuit to computean SHA1 value on a variable length string. In a non-limiting exemplaryembodiment of the transmission apparatus, the circuit to generate a dataparagraph, a hash, and a remainder comprises a circuit to compute an MD5value on a variable length string.

The method operates the circuit to generate a data paragraph, a hash,and a remainder by

computing a fingerprint; and

determining a data paragraph on the occurrence of a certain string beingmatched by a masked fingerprint or a maximum length being reached.

Alternately, in an embodiment, the method to generate a data paragraph,a hash, and a remainder operates by determining a data paragraph byusing digital signal processing detection of a repeating pattern.

The method operates within the circuit to generate a data paragraph, ahash, and a remainder by:

selecting a plurality of data paragraphs from the start of a dataobject; and

on the condition that no more data paragraphs can be selected, assigningthe rest of the data object as a remainder; and

transferring the hashes computed for each data paragraph and theremainder to a circuit for packing a data object integration packetaddressed to the reception apparatus.

The present patent application discloses a method for operating areception apparatus for receiving defragmented and de-duplicated dataobject packets, the apparatus comprising:

at least one network interface;

a primary cache store;

a primary transfer diary store; and

a data object integrator; the method comprising

receiving and storing into primary cache store at least one dataparagraph corresponding to a hash;

storing into primary transfer diary store the source of the paragraphcorresponding to the hash and the time of most recent usage;

receiving a data object integration packet containing at least one hashand a remainder; and

integrating a data object by reading data paragraphs from the primarycache store and concatenating the data paragraphs and the remainder.

The reception apparatus further comprises

a circuit to receive a pending transfer;

the circuit to receive a pending transfer coupled to a circuit totransmit a pending transfer;

the circuit to receive a pending transfer coupled to a primary transferdiary store; and

the circuit to receive a pending transfer coupled to a satellitetransfer diary store.

The method further comprises the steps

within a circuit to receive a pending transfer:

receiving a hash from a pending transfer transmitter corresponding to adataparagraph;

determining from a primary transfer diary store that the hash is alreadystored in a primary cache store; and

transmitting to a satellite transfer store an update on the status ofthe hash, whereby the satellite transfer diary store changes the statusof a pending transfer from pending to past and utilization of acommunications channel is optimized by de-duplication.

In an embodiment, the method further comprises the steps of checking themost recently used record of every data paragraph stored in cache store,and removing least recently used data paragraphs to provide availablestorage for new data. In an embodiment the method randomly removes dataparagraphs from cache store to provide available storage for new data.In an embodiment the invention includes the steps of storing dataparagraphs in cache store into a hierarchical directory and randomlyremoving an entire sub-directory to provide available storage for newdata.

In an embodiment, the method and system operate symmetrically, wherebythe primary apparatus receives a data object and transmits the hashes toat least one satellite apparatus.

In an embodiment, the method further comprises the step of synchronizingat least one satellite data paragraph cache store with the primary dataparagraph cache store in anticipation of future data transfers.

In an embodiment, the apparatus further comprises a data paragraph cachestore and a transfer diary store whereby data paragraphs transmittedfrom the apparatus can be reused in receiving incoming data objects.

In an embodiment, the apparatus further comprises a circuit forreceiving a plurality of packets, a circuit for combining the payload ofpackets to a data object, a circuit for computing fingerprints of thedata object to delimit data paragraphs, whereby a primary apparatus mayalso transmit data objects to at least one satellite apparatus.

CONCLUSION

The VPN defragmentation and de-duplication system advantageouslyprovides increased productivity, reduced IT costs, and enhanced dataintegrity and compliance. The VPN defragmentation and de-duplicationsystem improves application performance and data access in the branchoffice and central office because not every bit being sent to adestination travels over the communication network each time it isrequired.

Additionally, the claimed invention does not cache the data in thetraditional sense. Some data paragraphs may be retrieved locally even ifthe URL or filename for the data is different because the data paragraphis identified by a code and not by a URL or filename. Furthermore,unlike web caching, the claimed invention ensures that the data iscoherent by forwarding messages (e.g., data requests and responses)between the satellites and the primary. For example, web cachingoperates by locally intercepting messages for an authoritative source(e.g., a web server) and responding to the messages such that the webserver potentially never sees the messages. In some cases, particularlywith dynamic content, the locally cached copy may be stale orout-of-date. Advantageously, the VPN defragmentation and de-duplicationsystem provides the data coherency and up-to-date data by thetransparent operation of the transfer diary store.

Advantageously, the VPN defragmentation and de-duplication systemimproves application performance and data access. In some embodiments,by caching a data paragraph in both the satellite apparatus and theprimary apparatus, the VPN defragmentation and de-duplication systemminimizes the effects of latency and reduces network traffic over thecommunication network. Additionally, the content of the traffic isobfuscated by the insertion of codes which reference replaced dataparagraphs. Furthermore, accelerated access to the data paragraphslocally accessible to the satellite apparatus and the primary apparatusis not limited to a particular application or data context because acertain data paragraph may recur within a webpage, an email, and aspreadsheet shared among many authors and reviewers.

In an embodiment, the VPN defragmentation system includes a securetunnel between the satellite apparatus and the primary apparatus. Thesecure tunnel provides encryption (e.g., IPsec) and access control lists(ACLS). Additionally, in other embodiments, the secure tunnel includescompression, such as header and payload compression. The secure tunnelmay provide fragmentation/coalescing optimizations along with errordetection and correction.

The above-described functions can be comprised of executableinstructions that are stored on storage media. The executableinstructions can be retrieved and executed by a processor. Some examplesof executable instructions are software, program code, and firmware.Some examples of storage media are memory devices, tape, disks,integrated circuits, and servers. The executable instructions areoperational when executed by the processor to direct the processor tooperate in accord with the invention. Those skilled in the art arefamiliar with executable instructions, processor(s), and storage media.

The above description is illustrative and not restrictive. Manyvariations of the invention will become apparent to those of skill inthe art upon review of this disclosure. The scope of the inventionshould, therefore, be determined not with reference to the abovedescription, but instead should be determined with reference to theappended claims along with their full scope of equivalents.

1. A method for operating a transmission apparatus coupled to a localarea network and coupled to a wide area network, the transmissionapparatus configured with at least a circuit for to generate a dataparagraph, a hash, and a remainder, a satellite cache store, a satellitetransfer diary store, and a circuit for to pack a plurality of hashesand a remainder into a packet; the method comprising: receiving a dataobject; generating at least one data paragraph by selecting a stringbetween a minimum length and a maximum length by computing a fingerprintand matching selected bits of the fingerprint to a certain mask byreading from the beginning of a data object or from the end of theprevious data paragraph; computing a hash for each data paragraph;determining a remainder; storing a data paragraph into a primary cachestore when the hash is determined to be new; storing a record into asatellite transfer diary of the reception apparatus for each dataparagraph, the hash, and the status of the transfer of the dataparagraph to the reception apparatus; and transferring at least one hashand a remainder to a circuit to pack a data object integration packet.2. The method of claim 1 further comprising: reading a pending statusfor transfer of a data paragraph; transferring a data paragraph from aprimary cache store to a satellite apparatus; and updating the status ofthe record in satellite transfer diary store from pending to past. 3.The method of claim 2 further comprising: determining that all dataparagraphs required for data object integration are stored at thesatellite cache store; and enabling transmission of a data objectintegration packet to the reception apparatus.
 4. The method of claim 11further comprising: storing a data paragraph corresponding to each newhash; recording a status of transmission of each paragraph to eachreception apparatus; and packing a data integration packet with aremainder and at least one hash of a data paragraph for each dataobject.
 5. The method of claim 1 further comprising: synchronizing atleast one satellite data paragraph cache store with the primary dataparagraph cache store in anticipation of future data transfers.
 6. Themethod of claim 1 wherein computing a hash comprises: computing an SHA1value on a variable length string.
 7. The method of claim 1 whereincomputing a hash comprises: computing an MD5 value on a variable lengthstring.
 8. The method of claim 1 further comprising steps within thecircuit for to generate a data paragraph, a hash, and a remaindercomprising: computing a fingerprint; and determining a data paragraph onthe occurrence of one of a certain string being matched by a maskedfingerprint and a maximum length being reached.
 9. The method of claim 1further comprising steps within the circuit for to generate a dataparagraph, a hash, and a remainder comprising: determining a dataparagraph by using digital signal processing detection of a repeatingpattern.
 10. The method of claim 1 further comprising steps within thecircuit for to generate a data paragraph, a hash, and a remaindercomprising: selecting a plurality of data paragraphs from the start of adata object; and on the condition that no more data paragraphs can beselected; assigning the rest of the data object as a remainder; andtransferring the hashes computed for each data paragraph and theremainder to a circuit for packing a data object integration packetaddressed to the reception apparatus.